Information processing device, information processing method, and program

ABSTRACT

Conventional calculation of light source data requires a high load and is unstable. An information processing device derives light source data which represents a state of a light source of an image represented by a captured image based on image capturing condition data at the time of capturing the image of a subject.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to light source estimation.

2. Description of the Related Art

There is a case where image processing by image processing software etc.is performed in order to give the effect intended by a person who hascaptured an image to the captured image. For example, mention is made ofimage processing to change the kind and direction of a light source ofthe captured image, to synthesize a virtual object, etc. Such imageprocessing has required a skilled person to spend much time to create aprocessed image without unnaturalness, however, it has been madepossible to automatically perform image processing based on information,such as a subject shape, texture, a light source in an image capturingscene, etc. Further, various kinds of techniques to estimate suchinformation from a captured image have been proposed.

For example, there is a technique to estimate a light source byarranging a special apparatus, such as a two-dimensional marker providedwith a specular sphere in an image capturing scene. In this case, it ispossible to estimate the light source by performing calculationprocessing based on the position of a camera and the captured image ofthe specular sphere from which the light source is reflected. However,it is necessary to provide the special apparatus as described above andto capture an image after arranging the special apparatus in an imagecapturing scene, and therefore, there has been such a problem that mucheffort and time are required. In such circumstances, a method forestimating a light source by acquiring an entire periphery image by auser provided with a head mount display to which a video camera isattached moving about is proposed (Japanese Patent Laid-Open No.2008-33531).

There is also a technique to compare CG image data obtained by adjustinglight source parameters and rendering a subject shape under variouskinds of light source on condition that the shape and texture of thesubject be already known, and actually photographed image data, and toestimate a light source by which a difference therebetween is thesmallest as a light source in the real scene (Takahiro Okabe, “Sphericalharmonics vs. Haar wavelets: basis for recovering illumination from castshadows” Proc. IEEE Conf. Computer Vision and Pattern Analysis (CVPR04), pp. I-50-57, 2004, hereinafter referred to as “Okabe”).

With the method for estimating the light source according to JapanesePatent Laid-Open No. 2008-33531, it is necessary for the user to move inthe entire periphery direction in order to acquire the entire peripheryimage, and therefor, it is not possible to easily perform the method.

Further, in the case where the method for estimating the light source byOkabe is used, the number of light source parameters is large,therefore, there has been such a problem that optimization takes muchtime. Furthermore, there has been such a problem that the light sourceparameter results in a local solution, thereby the processing becomesunstable, as is known generally in the optimization question, in thecase where the difference between the CG image and the actuallyphotographed image is minimized.

SUMMARY OF THE INVENTION

An information processing device according to the present inventionincludes an image capturing condition data acquiring unit configured toacquire capturing an image condition data at the time of image capturingof a subject and a light source data deriving unit configured to derivelight source data which represents a state of a light source of theimage represented by the data of the captured image based on the imagecapturing condition data.

It is possible to estimate a light source in a real scene from capturedimage data easily, efficiently, and stably.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing an example of a configuration of a camera asan information processing device for performing light source estimationaccording to a first embodiment;

FIG. 2 is a function block diagram showing an internal configuration ofa light source estimating unit according to the first embodiment;

FIG. 3 is a diagram showing a relationship between a light source and asubject;

FIG. 4 is a diagram showing an example of light source data;

FIG. 5 is a flowchart showing a flow of light source estimationprocessing in the first embodiment;

FIG. 6 is a diagram showing an example of polygon data;

FIGS. 7A to 7D are diagrams showing examples of light source datacorresponding to image capturing condition data different from oneanother;

FIG. 8 is a diagram showing an example of a light source data table;

FIG. 9 is a flowchart showing a flow of light source data derivationprocessing in the first embodiment;

FIG. 10A is a diagram showing an example of a captured image and FIG.10B is a diagram showing data of a light source in the image capturingenvironment;

FIG. 11A is a diagram showing an example of a CG image having beensubjected to rendering and FIG. 11B is a diagram showing data of a lightsource used at the time of rendering;

FIG. 12 is a function block diagram showing an internal configuration ofa light source estimating unit according to a second embodiment;

FIG. 13 is a diagram showing an example of a configuration of a cameraincluding a plurality of image capturing units as an informationprocessing device for performing light source estimation according tothe second embodiment;

FIGS. 14A to 14C are diagrams each showing an example of amulti-viewpoint image in the case where the image is captured from threedifferent viewpoints;

FIG. 15 is a flowchart showing a flow of light source estimationprocessing in the second embodiment;

FIG. 16 is a diagram showing a system configuration example of aninformation processing system for performing light source estimationaccording to a third embodiment.

FIG. 17 is a flowchart showing a flow of light source estimationprocessing in the third embodiment;

FIG. 18 is a flowchart showing a flow of initial light source datadetermination processing in a processing server according to the thirdembodiment;

FIG. 19 is a diagram showing an example of a light source data tableaccording to the third embodiment;

FIG. 20 is a function block diagram showing an internal configuration ofa light source estimating unit according to a fourth embodiment;

FIG. 21 is a flowchart showing a flow of light source estimationprocessing in the fourth embodiment;

FIG. 22 is a function block diagram showing an internal configuration ofa light source estimating unit according to a fifth embodiment;

FIG. 23 is a flowchart showing a flow of light source estimationprocessing in the fifth embodiment;

FIG. 24 is a diagram showing an example of the light source data tableaccording to the present embodiment;

FIG. 25 is a function block diagram showing an internal configuration ofa light source estimating unit according to a sixth embodiment;

FIG. 26 is a flowchart showing a flow of light source estimationprocessing in the sixth embodiment;

FIG. 27 is a conceptual diagram of parallax image derivation in thesixth embodiment;

FIG. 28 is a conceptual diagram of conversion from a pixel deviationamount to a depth value in the sixth embodiment;

FIG. 29 is a detailed flowchart of step 2604 in the sixth embodiment;

FIG. 30A is a diagram showing an example of a CG rendering result in thecase where a captured image including only a subject that can beregarded as located at an infinite distance is used as light source dataand FIG. 30B is a diagram showing an example of a CG rendering result inthe case where a captured image including a subject that cannot beregarded as located at an infinite distance is used as light sourcedata;

FIG. 31 is a function block diagram showing an internal configuration ofa light source estimating unit according to an eighth embodiment;

FIG. 32 is a flowchart showing a flow of light source estimationprocessing in the eighth embodiment;

FIG. 33 is a diagram showing an example of an exclusionary region;

FIG. 34 is a diagram showing a state of a camera capturing an image in ascene in which a light irradiates a cuboid and a cube placed on a floor;

FIG. 35A shows a diagram showing an example of a captured image, FIG.35B is a diagram showing an example of light source data derived usingonly a cast shadow region of the captured image, and FIG. 35C is adiagram showing an example of light source data derived using the wholeregion except for the cast shadow region of a pixel region of thecaptured image;

FIG. 36A is a diagram showing an example of light source data derived inthe case where there is no exclusionary region and FIG. 36B is a diagramshowing an example of light source data derived in the case where thereis an exclusionary region;

FIG. 37 is a function block diagram showing an internal configuration ofa light source estimating unit according to a ninth embodiment;

FIG. 38 is a flowchart showing a flow of light source estimationprocessing in the ninth embodiment;

FIG. 39 is a diagram showing an example of a CG image generated byrendering;

FIG. 40 is a diagram for explaining a process of determination of aregion-to-be-excluded;

FIG. 41 is a diagram showing a relationship between a pixel position anda direction corresponding to the pixel position;

FIG. 42 is a diagram showing a system configuration example of aninformation processing system for performing light source estimationaccording to a tenth embodiment; and

FIG. 43 is a flowchart showing a flow of light source estimationprocessing in the tenth embodiment.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, preferred embodiments of the present invention areexplained with reference to the attached drawings.

First Embodiment

<Device Configuration>

FIG. 1 is a diagram showing an example of a configuration of a camera asan information processing device for performing light source estimationaccording to the present embodiment.

An image capturing unit 101 includes a zoom lens, a focus lens, a camerashake correction lens, a diaphragm, a shutter, an optical low-passfilter, an iR cut filter, a color filter, and sensors, such as CMOS andCCD, and detects quantity of light of a subject.

An A/D conversion unit 102 converts the quantity of light of the subjectinto a digital value.

A signal processing unit 103 performs white balance processing, gammaprocessing, noise reduction processing, etc. on a converted digitalvalue to generate digital image data.

A D/A conversion unit 104 performs analog conversion on generateddigital image data.

An encoder unit 105 performs processing to convert digital image datainto a file format, such as Jpeg and Mpeg.

A media interface 106 is an interface configured to connect to a PC andother medium (for example, hard disk, memory card, CF card, SD card, USBmemory). Further, the media interface 106 is connected to acommunication network, such as the Internet, and performs transmissionand reception of data in accordance with the necessity.

A CPU 107 is a processor that totally controls each unit.

A ROM 108 stores a control program etc. executed in the CPU 107.

A RAM 109 functions as a main memory, a work area, etc., of the CPU 107.

An image capturing system control unit 110 performs control of an imagecapturing system instructed from the CPU 107, such as focusing,releasing of the shutter, and adjustment of the diaphragm.

An operation unit 111 includes a button, a mode dial, etc., and receivesuser's instructions input therethrough.

A character generation unit 112 generates characters, graphics, etc.

As a display unit 113, generally, a liquid crystal display is usedwidely, and the display unit 113 displays captured images and charactersreceived from the character generation unit 112 and the D/A conversionunit 104. Further, it may also be possible for the display unit 113 tohave a touch screen function and in such a case, it is also possible tohandle user's instructions as an input of the operation unit 111.

A light source estimating unit 114 performs light source estimationprocessing from digital image data. Details of the light sourceestimating unit 114 will be described later.

A GPS receiver 115 acquires image capturing position data and suppliesthe data to the light source estimating unit 114 and the encoder unit105. Further, it is desirable for the GPS receiver 115 to have afunction as an electronic compass configured to acquire the direction ofa camera.

There are components of a camera other than those described above,however, they are not the main target of the present embodiment,therefore explanation thereof is omitted.

FIG. 2 is a function block diagram showing an internal configuration ofthe light source estimating unit 114 according to the presentembodiment.

The light source estimating unit 114 includes an image data acquiringunit 201, a subject shape data acquiring unit 202, an image capturingcondition data acquiring unit 203, a light source data deriving unit205, an initial light source data determining unit 204, and a lightsource data output unit 206.

The image data acquiring unit 201 acquires digital data of an image(hereinafter, simply referred to as “image data”) captured by the imagecapturing unit 101 and subjected to predetermined image processing fromthe signal processing unit 103.

The subject shape data acquiring unit 202 acquires shape data of asubject in the image relating to the image data from the ROM 108 etc.

The image capturing condition data acquiring unit 203 acquires datarelating to various kinds of condition at the time of image capturing(hereinafter, referred to as “image capturing condition data”) from theoperation unit 111, the image capturing system control unit 110, thesignal processing unit 103, the GPS receiver unit 115, etc. Imagecapturing condition data includes parameters indicative of a state andcircumstances for the camera at the time of image capturing.Specifically, shutter speed, whether or not flash is used, ISO speed,diaphragm stop, white balance (WB), date of image capturing, GPSinformation, image capturing mode, such as portrait and night scene,comment information, URL information, model type of the camera main bodyor lens used for image capturing, etc., are included in the imagecapturing condition data. Preferably, field angle of camera, resolution,position of optical center of camera, vector indicative of optical axisof camera, and information of vector indicative of upward direction ofcamera are included in the image capturing condition data. This imagecapturing condition data may be also directly input and set by a user,or may be recorded automatically to a header or a footer of image dataat the time of image capturing.

The initial light source data determining unit 204 determines initiallight source data corresponding to the image capturing condition data byreferring to various types of light source data (stored in the ROM 108etc. as a light source data storage unit in the state of beingassociated with image capturing condition data).

The light source data deriving unit 205 derives optimum light sourcedata as a light source in a scene from the image data, subject shapedata, image capturing condition data, and initial light source data.Here, light source data in the present embodiment is explained. FIG. 3is a diagram showing a relationship between a light source and asubject. In the present specification, it is assumed that a light sourcerepresents a luminance distribution for each incidence direction withrespect to light incident on a subject. The incidence direction isrepresented by a latitude θ and a longitude φ. FIG. 4 is a diagramshowing an example of light source data. In the example in FIG. 4, therow represents the longitude and the column represents the latitude, andnumerical values (0.0 to 1.0) indicative of luminance for each incidencedirection of light incident on a subject are shown. Such light sourcedata in which the values at 90 degrees latitude are extremely largerepresents a light source from a direction of a high point (that is,light source whose light intensity from the zenith is high). As a scenehaving such a light source, mention is made of a clear sky in thedaytime etc.

The light source data output unit 206 outputs the derived light sourcedata to outside.

Processing in each of the units described above is implemented by theCPU 107 causing various kinds of software (computer program) to run.

FIG. 5 is a flowchart showing a flow of light source estimationprocessing in the present embodiment.

At step 501, the image data acquiring unit 201 acquires image datahaving been subjected to predetermined image processing from the signalprocessing unit 103. The acquired image data is sent to the light sourcedata deriving unit 205.

At step 502, the subject shape data acquiring unit 202 acquires shapedata of a subject. The acquired subject shape data is sent to the lightsource data deriving unit 205. As subject shape data, polygon data ispreferable and data obtained by previously performing measurement usinga 3D scanner etc. and stored in the ROM 108 etc. in advance is acquired.Alternatively, it may also be possible to provide a 3D scanner in aninformation processing device itself and to directly acquire measuredpolygon data. FIG. 6 shows an example of polygon data acquired at thisstep, in which polygon data of a cube is shown. As shown in FIG. 6,polygon data includes vertex data and mesh data. In the case where thesubject is assumed to be a set of planes, a cube is a set of six planesand has eight vertexes. The vertex data holds a three-dimensionalcoordinate value of each vertex. The mesh data specifies each plane anddescribes how to configure a plane by connecting vertexes. For example,a plane S0 in the mesh data in FIG. 6 represents a square planespecified by vertexes v0, v1, v2, and v3. In FIG. 6, a plane isconfigured by four vertexes; however, it may also be possible toconfigure a plane by three vertexes or five vertexes. Such polygon datadescribing shape is acquired as subject shape data.

At step 503, the image capturing condition data acquiring unit 203acquires image capturing condition data from the operation unit 111 etc.The acquired image capturing condition data is sent to the initial lightsource data determining unit 204 and the light source data deriving unit205.

At step 504, the initial light source data determining unit 204determines initial light source data corresponding to the acquired imagecapturing condition data by referring to a light source data tablestored in the ROM 108 etc.

Here, a light source data table stored in the ROM 108 etc. is explained.In the ROM 108 etc. as a light source data storage unit, a light sourcedata table including a plurality of pieces of light source dataassociated with the image capturing condition data is held. Then, fromthe plurality of pieces of light source data, one piece of light sourcedata corresponding to the acquired image capturing condition data isdetermined as initial light source data. FIGS. 7A to 7D show examples oflight source data corresponding to different image capturing conditiondata, respectively. FIG. 7A is an example of light source data suitableto the case where the setting of white balance is “cloudy”. In the casewhere white balance is “cloudy”, it is inferred that the image capturingscene is a cloudy or shady scene, and therefore, the light source at thetime of image capturing is uniform in all the directions. Because ofthis, light source data in which luminance has a uniform value in allthe directions is preferable. FIG. 7B is an example of light source datasuitable to the case where the setting of white balance is “sunlight”.In the case where white balance is “sunlight”, it is inferred that theimage capturing scene is a clear sky scene. Because of this, lightsource data in which luminance of light from the zenith direction ishigh is preferable. In the case where white balance is “sunlight”, it ismore desirable to predict the accurate direction of the sun based oninformation, such as GPS information and date of image capturing, and tohold light source data in which luminance is highest in the predicteddirection. FIG. 7C is an example of light source data suitable to theuse of flash. The probability that flash is irradiated in the horizontaldirection is high. Because of this, light source data in which luminancein the 0° latitude direction is highest is preferable. FIG. 7D is anexample of light source data suitable to the case where the setting ofimage capturing mode is “night scene”. In this case, it is inferred thatthe image capturing scene is a night scene. Because of this, lightsource data in which luminance in the horizontal direction iscomparatively high due to the light of a town is preferable.

Such various types of light source data are held in the ROM 108 etc. inthe state of being associated with the image capturing condition data.FIG. 8 is an example of a light source data table holding a plurality ofpieces of light source data in the state of being associated with theimage capturing condition data. The initial light source datadetermining unit 204 searches initial light source data suitable to theimage capturing condition data at that time by referring to such a lightsource data table. Specifically, an index E is calculated using thefollowing formula (1) for each row of the light source data table shownin FIG. 8 and initial light source data is specified so that the index Ewill be a minimum.

$\begin{matrix}{\mspace{79mu} {{E = {\sum{w_{i}\left( {v_{i} - a_{i}} \right)}^{2}}}\mspace{79mu} \text{?}{\text{?}\text{indicates text missing or illegible when filed}}}} & \left\lbrack {{Formula}\mspace{14mu} (1)} \right\rbrack\end{matrix}$

Where, v_(i) and a_(i) are parameters of the image capturing conditiondata, wherein v_(i) is the image capturing condition data acquired atstep 503 and a_(i) is the image capturing condition data in the tableheld by the ROM 108 etc. The subscript corresponds to the number ofparameters of the image capturing condition data and, for example, v₁/a₁is the longitude of GPS information, v₂/a₂ is the latitude of GPSinformation, v₃/a₃ is the date of image capturing, and so on. Withregard to parameters not represented by a numerical value, such as WBsetting, an arbitrary numerical value shown in brackets in FIG. 8 isallocated etc. (for example, to the WB sunlight: a numerical value “0”)is allocated, so as to bring about a state where comparison can be made.A coefficient w_(i) is a weight coefficient. In the case where it isintended to use GPS information as a main determination factor withwhich to determine initial light source data, it is preferable to setthe weight coefficient for GPS information larger than those for otherparameters.

In the manner described above, light source data corresponding to theimage capturing condition data acquired at step 503 is determined asinitial light source data.

At step 505, the light source data deriving unit 205 derives lightsource data optimum for a scene based on the acquired image data,subject shape data, image capturing condition data (steps 501 to 503),and the determined initial light source data (step 504). Details oflight source data derivation processing will be described later.

At step 506, the light source data output unit 206 outputs the derivedlight source data.

(Light Source Data Derivation Processing)

FIG. 9 is a flowchart showing a flow of light source data derivationprocessing at step 505. By this processing, optimized light source databy which it is possible to obtain a CG image close to the actuallycaptured image.

At step 901, the light source data deriving unit 205 sets initial lightsource data sent from the initial light source data determining unit 204as an initial value of light source data to be found. As will bedescribed later, this flowchart has a loop structure and light sourcedata in an i-th loop is represented by L_(i) (θ, φ). At step 901, as aninitial value, L₀ (θ, φ) is set.

At step 902, the light source data deriving unit 205 performs renderingbased on the initial light source data set at step 901 and the subjectshape data and the image capturing condition data acquired at steps 502and 503 described previously to generate CG image data. The imagecapturing condition data here is, specifically, parameters necessary forrendering, such as the field angle and the position of the camera. Asthe rendering method, a method aiming at accuracy physically, such aspath tracing and photo mapping, is preferable so that a CG image and anactually photographed image can be compared. For details of this step,see [Okabe]. In addition, it is assumed that a CG image subjected torendering using the light source data L_(i) (θ, φ) is represented asI_(i). Here, with reference to FIG. 10 and FIG. 11, the relationshipbetween light source data and a CG image is explained. FIG. 10A is anexample of a captured image (actually photographed image) and FIG. 10Bshows data of the light source in the image capturing environment (thatis, the light source data to be estimated). As is obvious from FIG. 10B,in this case, light is irradiated only from the 45° latitude and 0°longitude direction, and therefore it is possible to see a cast shadowin the position obliquely right under the cuboid, which is the subjectin the captured image shown in FIG. 10A. On the other hand, FIG. 11A isan example of a CG image having been subjected to rendering and FIG. 11Bshows data of the light source used at the time of the rendering. As isobvious from FIG. 11B, light is irradiated from an erroneous direction(the latitude is correct, however, the longitude is not 0° but 90°), andtherefore it is possible to see a cast shadow in the position obliquelyleft under the cuboid, which is a subject, in the rendering result (CGimage) shown in FIG. 11A. From the examples shown in FIG. 10 and FIG.11, it is known that the light source data shown in FIG. 11B isinappropriate as light source data in the scene because the differencebetween the actually captured image and the CG image having beensubjected to rendering is large. For the convenience of explanation, theexample is used in which the light source data used for rendering isextremely inappropriate; however, in the light source data derivationprocessing according to the present embodiment, as explained already,light source data comparatively close to the ideal is set as initiallight source data.

At step 903, the light source data deriving unit 205 finds an errorΔ_(i) between the CG image generated at step 902 and the captured image,and determines whether the error Δ_(i) that is found is smaller than apredetermined threshold value. As a method for finding an error, it mayalso be possible to find the RMS error, which requires a light load ofcalculation or to use the VDP (Visual Difference Predictor) forcalculating a difference between images by making use of the humanvisual characteristics or the S-CIELAB. In the case where the errorΔ_(i) that is found is smaller than a predetermined threshold value (forexample, in the case where the S-CIELAB is used to find an error, 1,which is the human detection limit, is set as the threshold value), theprocedure proceeds to step 905. On the other hand, in the case where theerror Δ_(i) that is found is larger than the predetermined value, theprocedure proceeds to step 904.

At step 904, the light source data deriving unit 205 updates thecontents of the light source data based on the error Δ_(i) found at step903. Specifically, next light source data L_(i+1) (θ, φ) is calculatedusing the following formula (2).

$\begin{matrix}{\mspace{79mu} {{{L_{i + 1}\left( {\theta,\phi} \right)} = {{L_{i}\left( {\theta,\phi} \right)} - {\frac{\left( {\Delta_{i} - \Delta_{i - 1}} \right)}{{L_{i}\left( {\theta,\phi} \right)} - {L_{i - 1}\left( {\theta,\phi} \right)}}\delta}}}\mspace{79mu} \text{?}{\text{?}\text{indicates text missing or illegible when filed}}}} & \left\lbrack {{Formula}\mspace{14mu} (2)} \right\rbrack\end{matrix}$

Where, δ is a parameter to specify an extent to which the light sourcedata is updated and for example, is set to 1. In the case where muchtime is required for calculation, δ is set to a larger value or on thecontrary, in the case where the set value is too large and precision islow, δ is set to a smaller value, thus δ is set appropriately. At thepoint of time of the first loop (that is, in the case where i=0),neither Δ_(i−1) nor (θ, φ) is obtained yet. Consequently, in the firstloop, in place of the formula (2), the following formula (3) is used toupdate light source data.

L _(i+1)(θ,φ)=L _(i)(θ,φ)−δ  [Formula (3)]

After the light source data is updated in this manner and the nextupdated data L_(i+1) (θ, φ) is obtained, the procedure returns to step902 and rendering is performed based on the updated light source data.Then, whether the error Δ_(i) between a newly obtained CG image and thecaptured image is smaller than the threshold value is determined (step903) and the same processing is repeated until it is determined that theerror Δ_(i) is smaller than the threshold value.

At step 905, the light source data deriving unit 205 outputs the derivedlight source data (the light source data for which it is determined thatthe error Δ_(i) is smaller than the predetermined threshold value) tothe light source data output unit 206.

In this manner, light source data optimum for a scene is derived.

As described above, according to the invention of the presentembodiment, in the processing to derive appropriate light source data,the initial value of light source data (initial light source data) isset based on the image capturing condition data which represents thecharacteristics of the light source, such as the white balance setting.Due to this, it is possible to cause optimized calculation of lightsource data to converge earlier and further, to reduce a risk that anerroneous result is obtained because the optimization results in a localsolution.

In the present embodiment, a camera is used as an example of aninformation processing device for performing light source estimation;however, this is not limited. For example, it may also be possible for aCPU of a computer having received digital image data to perform lightsource estimation processing by causing various kinds of software torun.

Second Embodiment

In the first embodiment, the subject shape data obtained by performingmeasurement using the 3D scanner etc. and stored in the ROM 108 etc. inadvance is acquired and used for light source data derivationprocessing. Next, an aspect is explained as a second embodiment, inwhich estimation of subject shape is performed based on acquired imagedata and image capturing condition data, and light source data isderived using the obtained shape data. Explanation of parts common tothose of the first embodiment is simplified or omitted and here,different points are explained mainly.

FIG. 12 is a function block diagram showing an internal configuration ofthe light source estimating unit 114 according to the presentembodiment. The light source estimating unit 114 differs greatly fromthe light source estimating unit 114 according to the first embodiment(see FIG. 2) in that a subject shape estimating unit 1201 is provided inplace of the subject shape data acquiring unit 202.

In the case of the present embodiment, it is preferable that image dataacquired by the image data acquiring unit 201 is data which consists ofimages captured from a plurality of different viewpoints(multi-viewpoint image data). FIG. 13 is a diagram showing an example ofa configuration of a camera, as an information processing device forperforming light source estimation, including a plurality of imagecapturing units according to the present embodiment. FIG. 13 shows twoimage capturing units, that is, image capturing units 1301 and 1302;however, it is possible to arbitrarily set the number of image capturingunits and arrangement thereof. For example, it is possible to consider acase where nine image capturing units are arranged uniformly on a squarelattice and in the case of such a camera adopting a camera array system,it is possible to obtain data of images from nine viewpoints by one-timeimage capturing. Further, it may also be possible to obtainmulti-viewpoint image data by performing image capturing a plurality oftimes while shifting the viewpoint using a camera having a single lenssystem image capturing unit configured to capture one image by one-timeimage capturing (see FIG. 1). FIGS. 14A to 14C are diagrams each showingan example of a multi-viewpoint image in the case where the images arecaptured from three different viewpoints using, for example, a camerahaving a single lens system image capturing unit.

FIG. 15 is a flowchart showing a flow of light source estimationprocessing in the present embodiment.

At step 1501, the image data acquiring unit 201 acquires multi-viewpointimage data having been subjected to predetermined image processing fromthe signal processing unit 103. The acquired multi-viewpoint image datais sent to the subject shape estimating unit 1201 and the light sourcedata deriving unit 205.

At step 1502, the image capturing condition data acquiring unit 203acquires image capturing condition data from the operation unit 111 etc.The acquired image capturing condition data is sent to the subject shapeestimating unit 1201, the initial light source data determining unit204, and the light source data deriving unit 205.

At step 1503, the subject shape estimating unit 1201 estimates the shapeof a subject from multi-viewpoint image data. The method for estimatingthe shape of a subject from a plurality of images with differentviewpoints is widely known, and for example, it is possible to performcalculation using the SfM (Structure from Motion). Although explanationof specific estimation processing is omitted here, it is preferable toinclude position information etc. of the camera in the image capturingcondition data and to make use thereof in the case where, for example,the SfM is adopted. The calculated subject shape data is sent to thelight source data deriving unit 205 together with an evaluation valuewhich represents accuracy or reliability of the data. Here, theevaluation value is explained. In shape estimation, points (regions)corresponding to each other between a plurality of images are extracted.Then, an evaluation value is given to subject shape data in such amanner that the higher the degree of coincidence between correspondingpoints, the higher the evaluation value is. At this time, it ispreferable to take a threshold value (for example, 90%) set in relationto the degree of coincidence to be an evaluation value of shapeestimation.

At step 1504, the initial light source data determining unit 204determines initial light source data corresponding to the imagecapturing condition data by referring to alight source data table savedin the ROM 108 etc. The processing here is the same as that at step 504of the flowchart in FIG. 5 according to the first embodiment.

At step 1505, the light source data deriving unit 205 derives lightsource data in the scene based on the acquired multi-viewpoint imagedata and image capturing condition data, the calculated subject shapedata, and the determined initial light source data. The light sourcedata derivation processing according to the present embodiment issubstantially the same as that at step 505 of the flowchart in FIG. 5according to the first embodiment (see the flowchart in FIG. 9),however, differs in the following points.

First, the multi-viewpoint image data is acquired at step 1501,therefore the rendering processing at step 902 and the errordetermination processing of a CG image and a captured image at step 903are also performed for each of the images with different viewpoints(that is, nine times in the case where images corresponding to nineviewpoints are included).

Further, the threshold value at the time of evaluation of the errorbetween the CG image and the captured image at step 903 is set asfollows. In the case of the subject shape data estimated from aplurality of images, as a matter of course, the deviation from theactual subject is usually large compared to the subject shape data etc.obtained by using a measurement device, such as a 3D scanner.Consequently, it is supposed that the error between the CG image havingbeen subjected to rendering using subject shape data with a largedeviation, and the actually captured image is also large. Because ofthis, the threshold value used at this step is set to a value largerthan that in the case of the first embodiment. Specifically, thethreshold value at step 903 is set in accordance with the evaluationvalue of shape estimation received together with the subject shape data.For example, at the time where correspond points are extracted in thesubject shape estimating unit 1201, it is assumed that an algorithmadopts as a corresponding point in the case where a difference betweenimages near the corresponding point is equal to or less than a thresholdvalue C. In this case, the threshold value at step 903 is also set to C,and so on. It is desirable to appropriately set the threshold value atstep 903 depending on the magnitude of the error that is supposed, forexample, to set the threshold value to 2C in the case where an erroroccurs by another factor.

At step 1506, the light source data output unit 206 outputs the derivedlight source data. As described above, in the case of the presentembodiment, the light source data, for which it is determined that theerror is smaller than the threshold value, is derived in the number ofpieces of data corresponding to the number of images with differentviewpoints, and therefore, there is a possibility that light source datawhose contents are different for each image is derived depending on theset threshold value. In such a case, it is sufficient to determine onepiece of light source data by repeating the processing at step 1505until the same result is obtained for all the images by setting thethreshold value again, or taking the light source data, for which it isdetermined that the error is smaller than the threshold value in thelargest number of images, to be the final light source data, or thelike.

As described above, according to the invention of the presentembodiment, the shape of a subject is estimated from multi-viewpointimage data, therefore it is possible to obviate the need to preparesubject shape data by measuring the shape of the subject using a 3Dscanner etc. in advance.

Third Embodiment

Next, an aspect is explained as a third embodiment, in which initiallight source data used for light source data derivation processing isdetermined by using an external server etc. connected to a communicationnetwork. Explanation of parts common to those of the other embodimentsis simplified or omitted and here, different points are explainedmainly.

FIG. 16 is a diagram showing a system configuration example of aninformation processing system for performing light source estimationaccording to the present embodiment. A light source estimation device1600 is a PC etc. and substantially corresponds to the light sourceestimating unit 114 in the first embodiment. The light source estimationdevice 1600 according to the present embodiment differs largely from thelight source estimating unit 114 according to the first embodiment (seeFIG. 2) in that the light source estimation device 1600 does not includethe initial light source data determining unit but includes an imagecapturing condition data transmitting unit 1604 and an initial lightsource data receiving unit 1605 instead. Then, the light sourceestimation device 1600 is connected to a processing server 1610 as aninitial light source data determining unit and a data server 1620 as alight source data storage unit (light source database) to each other viaa communication network, such as the Internet.

FIG. 17 is a flowchart showing a flow of light source estimationprocessing in the present embodiment.

At step 1701, an image data acquiring unit 1601 of the light sourceestimation device 1600 acquires image data having been subjected topredetermined image processing from a camera etc., not shownschematically. The acquired image data is sent to a light source dataderiving unit 1606.

At step 1702, a subject shape data acquiring unit 1602 of the lightsource estimation device 1600 acquires subject shape data from an HDD(not shown schematically) etc. in which subject shape data is stored.

At step 1703, an image capturing condition data acquiring unit 1603 ofthe light source estimation device 1600 acquires image capturingcondition data from a camera etc., not shown schematically. The acquiredimage capturing condition data is sent to the light source data derivingunit 1606 and the image capturing condition data transmitting unit 1604.

At step 1704, the image capturing condition data transmitting unit 1604transmits the acquired image capturing condition data to the processingserver 1610 as an initial light source data determining unit.

At step 1705, the processing server 1610 determines one piece of initiallight source data corresponding to the image capturing condition datafrom various pieces of light source data stored in the data server 1620and transmits the data to the light source estimation device 1600.

FIG. 18 is a flowchart showing a flow of initial light source datadetermination processing in the processing server 1610 according to thepresent embodiment.

At step 1801, the processing server 1610 receives image capturingcondition data from the light source estimation device 1600.

At step 1802, the processing server 1610 adds light source relatedinformation to the received image capturing condition data. The lightsource related information refers to, for example, weather informationat the point of image capturing obtained from GPS information, date,etc., included in the image capturing condition data. The weatherinformation may be information indicative of clear or cloudy weather, orinformation of an image of the image capturing point captured from thesky by a weather satellite at the time of image capturing (or before orafter the time). The use of a captured image has an advantage that moredetailed weather information can be obtained. Such weather informationis acquired by making use of the weather services etc. provided by aprocessing device connected to the communication network. As analternative example of light source related information, there can bethought, for example, information, such as the name of a buildinglocated at the image capturing point, which is obtained by acquiring theaddress of the image capturing point using services, such as inverseGeocoding, from GPS information.

At step 1803, the processing server 1610 accesses the data server 1620,searches for light source data corresponding to the image capturingcondition data, and determines initial light source data based on theimage capturing condition data to which light source related informationis added. The method for determining initial light source data is thesame as that at step 504 of the first embodiment.

At step 1804, the processing server 1610 transmits the determinedinitial light source data to the light source estimation device 1600.

Explanation is returned to the flowchart in FIG. 17.

At step 1706, the initial light source data receiving unit 1605 of thelight source estimation device 1600 receives the initial light sourcedata.

At step 1707, the light source data deriving unit 1606 derives lightsource data. The method for deriving light source data is the same asthat explained in the flowchart in FIG. 9 of the first embodiment.

At step 1708, a light source data output unit 1607 outputs the derivedlight source data. Outputting here includes saving the light source datain a storage medium, for example, a memory card etc., not shownschematically. Further, the derived light source data after calculationfor optimization and the image capturing condition data thereof are sentto the data server 1620. At this time, it is preferable to transmit alsoan evaluation value (value indicative of the degree of matching with theactual light source) of the derived light source data. The evaluationvalue may be, for example, the threshold value used at step 903 of theflowchart in FIG. 9 or a value (0 to 100%: the larger the value, thehigher the evaluation is) a user sets separately. In the data server1620 having received the light source data and image capturing conditiondata thereof, processing to add the received light source data and imagecapturing condition data thereof to a light source data table(processing to update a light source data table) is performed. FIG. 19shows an example of the updated light source data table, to whichweather information 1901 and an evaluation value 1902 are added, whichare items not included in the light source data table (see FIG. 8)according to the first embodiment. Preferably, maker side creates alight source data table including several kinds of basic light sourcedata at first as the light source database according to the presentembodiment, and then, the light source data table is updated each time auser creates light source data afterward (or in response to instructionsfor update from a user at that time).

As described above, according to the invention of the present invention,an independent light source database shared on a communication networkis used, and therefore, it is possible to hold more pieces of lightsource data and to quickly and stably derive light source data mostsuitable to a scene. Further, newly created light source data is addedto the light source database each time it is created, and therefore,derivation of light source data with higher precision is enabled.

Fourth Embodiment

Next, an aspect is explained as a fourth embodiment, in which lightsource data is derived easily without using shape data of an actualsubject. Explanation of parts common to those of the other embodimentsis simplified or omitted and here, different points are explainedmainly.

FIG. 20 is a function block diagram showing an internal configuration ofthe light source estimating unit 114 according to the presentembodiment.

The light source estimating unit 114 according to the present embodimentincludes the image data acquiring unit 201, the image capturingcondition data acquiring unit 203, the light source data deriving unit205, the initial light source data determining unit 204, and the lightsource data output unit 206, and a subject shape data acquiring unit 202configured to acquire shape data of an actual subject is not included.

FIG. 21 is a flowchart showing a flow of light source estimationprocessing in the present embodiment.

At step 2101, the image data acquiring unit 201 acquires image datahaving been subjected to predetermined image processing from the signalprocessing unit 103. The acquired image data is sent to the light sourcedata deriving unit 205.

At step 2102, the image capturing condition data acquiring unit 203acquires image capturing condition data from the operation unit 111 etc.The acquired image capturing condition data is sent to the light sourcedata deriving unit 205 and the initial light source data determiningunit 204.

At step 2103, the initial light source data determining unit 204determines initial light source data corresponding to the imagecapturing condition data by referring to a light source data table savedin the ROM 108 etc.

At step 2104, the light source data deriving unit 205 derives lightsource data of a scene based on the acquired image data, the imagecapturing condition data (steps 2101, 2102) and the determined initiallight source data (step 2103). In the light source data derivationprocessing in the present embodiment, shape data of a subject is notinput, therefore rendering is performed using simplified shape data inplace thereof. Here, as simplified shape, for example, a plane havingthe material attribute of a reflectance of 50% that stands facing acamera is preferable. It is desirable to set the distance between theplane and the camera in this case to a subject distance included in theimage capturing condition data. Further, it is desirable to set the sizeof the plane to the size of a human, such as 2 m×0.5 m in the case ofthe portrait mode, or to a size that covers the whole of the image fromthe field angle of the camera included in the image capturing conditiondata in the case of other than the portrait mode. Data of simplifiedshape in place of shape data of an actual subject is provided in an HDDetc. in advance. By rendering based on such data of simplified shape, aCG image is generated and optimum light source data is derived using theCG image data.

At step 2105, the light source data output unit 206 outputs the derivedlight source data.

It is also possible to apply the method that uses simplified shape datadescribed above to the third embodiment and to derive light source datavia a communication network. At that time, it is desirable to attach anindex indicating that the light source data is derived in the statewhere there is no subject shape data to the light source data in thelight source database.

As described above, according to the invention of the presentembodiment, simplified shape data is used instead of acquiring subjectshape data, and therefore, it is possible to derive optimum light sourcedata more quickly.

Fifth Embodiment

In the fourth embodiment, light source data is derived using simplifiedshape data in place of subject shape data. Next, an aspect is explainedas a fifth embodiment, in which light source data is derived more easilywithout using even simplified shape data. Explanation of parts common tothose of the other embodiments is simplified or omitted and here,different points are explained mainly.

FIG. 22 is a function block diagram showing details of the light sourceestimating unit 114 according to the present embodiment. The lightsource estimating unit 114 according to the present embodiment includesthe image data acquiring unit 201, the image capturing condition dataacquiring unit 203, a feature quantity deriving unit 2201, the lightsource data deriving unit 205, and the light source data output unit206. The configuration unique to the present embodiment is the featurequantity deriving unit 2201.

The feature quantity deriving unit 2201 analyzes image data receivedfrom the image data acquiring unit 201 and derives feature quantities,which are features of the image represented by numerical values(hereinafter, referred to as “image feature quantities”), such as anaverage color temperature of the image, a histogram of the pixel values,an average Lab value of light source data, and a variance value of thehistogram of the pixel values. In the present embodiment, a case wherethe average color temperature of the image and the histogram of thepixel values are adopted as the image feature quantities is explained.

FIG. 23 is a flowchart showing a flow of light source estimationprocessing in the present embodiment.

At step 2301, the image data acquiring unit 201 acquires image datahaving been subjected to predetermined image processing from the signalprocessing unit 103. The acquired image data is sent to the featurequantity deriving unit 2201.

At step 2302, the image capturing condition data acquiring unit 203acquires image capturing condition data from the operation unit 111 etc.The acquired image capturing condition data is sent to the light sourcedata deriving unit 205.

At step 2303, the feature quantity deriving unit 2201 analyzes the imagedata received from the image data acquiring unit 201 and derives imagefeature quantities (here, the average color temperature of the image andthe histogram of the pixel values). Specifically, derivation isperformed as follows.

In the case of the average color temperature of the image, first, RGBvalues are converted into chromaticities XYZ based on the used colorspace profile of the image and the chromaticities are converted intocorrelated color temperatures, thereby the color temperatures of theimage are obtained. Then, the obtained color temperatures are averagedto derive the average color temperature.

In the case of the histogram of the pixel values, the number of times ofappearance of the luminance value of each pixel in the image is countedand the frequency of appearance is found, thereby the histogram isderived.

At step 2304, the light source data deriving unit 205 refers to a lightsource data table saved in the ROM 108 etc. and acquires light sourcedata corresponding to the image capturing condition data and the imagefeature quantities. FIG. 24 is a diagram showing an example of the lightsource data table according to the present embodiment. In the case ofthe light source data table in the present embodiment, a plurality ofpieces of light source data is held in the state of being associatedwith the image capturing condition data and the feature quantities ofthe light source data corresponding to the image feature quantities.Here, as the feature quantities of the light source data, information ofthe average color temperature and the histogram of the pixel values areassociated and held. The average color temperature of the light sourcedata is obtained as follows. First, the luminance value held for eachangle is extended to the RGB value. Then, the RGB value corresponding toeach direction is converted into the color temperature. The convertedcolor temperature is averaged. The histogram is obtained as follows. Thenumber of times of appearance of each luminance (0 to 1) in the lightsource data is counted at 0.1 intervals by setting bins and the countednumber is divided by the number of samples. For example, in the case ofthe light source data shown in FIG. 7D, the counted number of eachluminance is (0, 0, 8, 0, 4, 0, 0, 0, 0) and the counted number isdivided by 12, the number of samples, thereby (0, 0, 0.66, 0, 0.33, 0,0, 0, 0) is obtained as the histogram. It is needless to say the in thecase where an index other than the average color temperature and thehistogram of the pixel values is used as an image feature quantity, avalue corresponding to the image feature quantity is found and held inthe light source data table.

A method for creating a histogram of image data that can be comparedwith the histogram of light source data, such as that represented by (0,0, 0.66, 0, 0.33, 0, 0, 0, 0) described above is shown below. It isassumed that the pixel values of image data are set in the range between0 and 255. First, each pixel in the image data is converted so that therange of the pixel values is between 0 and 1. Such conversion can beimplemented by dividing each pixel value of each pixel by 255. Next,bins are set at 0.1 intervals in the range between 0 and 1 and thenumber of times of appearance of each pixel value of each pixel afterthe conversion is counted. As a data example of the count result,mention is made of (5000, 10000, 650000, 320000, 10000, 5000, 0, 0) etc.in the case where the image size is 1,000 pixels×1,000 pixels. The dataof the count result is divided by the number of samples, that is,1,000×1,000=1,000,000, thereby (0.005, 0.01, 0.65, 0.32, 0.01, 0.005, 0,0) is obtained. This data is the histogram of the image data that can becompared with the histogram of the light source data.

At this step, by referring to the light source data table as describedabove, one piece of light source data corresponding to the imagecapturing condition data acquired at step 2302 and the image featurequantities derived as step 2303 is selected. It is possible to performthis selection by the same method as that at step (step 504) at whichthe initial light source data is determined of the flowchart in FIG. 5according to the first embodiment. That is, the index E is calculatedusing the formula (1) described previously for each row of the lightsource data table as shown in FIG. 24 and light source data is selectedso that the index E becomes a minimum.

At step 2305, the light source data deriving unit 205 outputs thederived light source data to the light source data output unit 206. Asthe output method in this case, it may also be possible to record thelight source data at the header of the image data to be saved or togenerate and save a light source data file separately from the imagefile.

As described above, according to the invention of the presentembodiment, it is possible to easily derive light source data close tothe light source in the environment in which image capturing isperformed.

Sixth Embodiment

Next, an aspect is explained as a sixth embodiment, in which lightsource data is derived by processing data of a captured image.Explanation of parts common to those of the other embodiments issimplified or omitted and here, different points are explained mainly.

FIG. 25 is a function block diagram showing details of the light sourceestimating unit 114 according to the present embodiment. The lightsource estimating unit 114 according to the present embodiment includesthe image data acquiring unit 201, the image capturing condition dataacquiring unit 203, a depth image generating unit 2501, the light sourcedata deriving unit 205, and the light source data output unit 206. Theconfiguration unique to the present embodiment is the depth imagegenerating unit 2501.

The depth image generating unit 2501 generates a depth imagerepresenting information of a distance from a camera to a subject by animage. Details of depth image generation will be described later.

It is preferable for image data acquired by the image data acquiringunit 201 according to the present embodiment to be multi-viewpoint imagedata of images captured from a plurality of different viewpoints as inthe second embodiment.

FIG. 26 is a flowchart showing a flow of light source estimationprocessing in the present embodiment.

At step 2601, the image data acquiring unit 201 acquires multi-viewpointimage data having been subjected to predetermined image processing fromthe signal processing unit 103. The acquired multi-viewpoint image datais sent to the depth image generating unit 2501 and the light sourcedata deriving unit 205.

At step 2602, the image capturing condition data acquiring unit 203acquires image capturing condition data from the operation unit 111 etc.The acquired image capturing condition data is sent to the depth imagegenerating unit 2501 and the light source data deriving unit 205.

At step 2603, the depth image generating unit 2501 generates a depthimage from the multi-viewpoint image data received from the image dataacquiring unit 201 and the image capturing condition data correspondingto each of the plurality of captured images with different viewpoints.The widely known methods may be used to generate a depth image andmention is made of, for example, the method described in the fourthmodification example in Japanese Patent Laid-Open No. 2009-165115. Here,a case where a depth image is generated from two images is explained asan example.

At the time of generation of a depth image, first, a parallax image isgenerated based on two captured images with different viewpoints. FIG.27 is a diagram showing a way a parallax image is generated from twocaptured images with different viewpoints. In each of an image A and animage B, there are a person, a tower, and buildings as subjects. First,a region a predetermined size (for example, 11×11 pixels) is set foreach pixel position of the captured image A and a region having highcorrelation with the region is searched for from the captured image B.Here, the pixel position at the center of the region in the capturedimage B having high correlation with the region whose center is a pixelposition (xA, yA) of the captured image A is taken to be (xB, yB). Inthe case where the subject is at a finite distance, (xA, yA) and (xB,yB) do not agree with each other and a deviation occurs. It is knownthat this pixel deviation becomes smaller as the distance of the subjectincreases. Consequently, a parallax image is obtained by calculating apixel deviation amount dx and by allocating the calculated pixeldeviation amount dx to each pixel position of the captured image A. Inthis case, the pixel deviation amount dx is expressed by the followingformula.

[Formula (4)]

In the parallax image shown in FIG. 27, the person whose pixel deviationamount dx is the largest is represented in white, the tower andbuildings whose pixel deviation amount dx is the second largest arerepresented in light gray, and the background whose pixel deviationamount dx is the smallest is represented in dark gray.

After parallax image data is obtained, the parallax image data isconverted into depth image data based on the image capturing conditiondata. The method for converting a pixel deviation amount into a distancein accordance with the cameral characteristic values is widely known;therefore, a case of simple geometry is explained. In FIG. 27 describedpreviously, it is assumed that the subject located at the center of thecaptured image A is located in a position deviated by the pixeldeviation amount dx in the horizontal direction in the captured image B.Here, a number M of horizontal pixels and a field angle θ of the cameraare acquired from the image capturing condition data and further, adistance D between the optical center of the captured image A and theoptical center of the captured image B is derived from the position dataof the camera. Then, in accordance with the following formula (5), thepixel deviation amount dx is converted into a distance L (depth value)to the subject.

[Formula (5)]

FIG. 28 is a diagram showing that the distance (depth value) to thesubject is derived by the formula (5) described above.

In this manner, the depth image corresponding to the captured image A isgenerated. In the above-described explanation, it is needless to saythat a depth image corresponding to the captured image B is obtained byperforming the processing after exchanging the roles of the capturedimage A and the captured image B.

In the manner described above, from multi-viewpoint image data and imagecapturing condition data, a depth image corresponding to each ofcaptured images with different viewpoints is generated.

Explanation is returned to the flowchart in FIG. 26.

At step 2604, the light source data deriving unit 205 derives lightsource data from multi-viewpoint image data, image capturing conditiondata, and depth image data.

FIG. 29 is a flowchart showing a flow of light source data derivationprocessing in the present embodiment.

At step 2901, the light source data deriving unit 205 initializes lightsource data. Specifically, the light source data deriving unit 205 setsa numerical value (for example, −1) etc. indicative of an unset statefor the luminance values corresponding to all the latitudes andlongitudes of the light source data.

At step 2902, the light source data deriving unit 205 sets a capturedimage to be subjected to processing from the multi-viewpoint image data.

At step 2903, the light source data deriving unit 205 sets a pixelposition n that is referred to in the captured image set to be subjectedto processing. For example, in the case where the pixel position at thetop-left end of the image is set at first in the stage immediately afterthe start of processing, the pixel position n is sequentially updated to(n+1) and so on from the top-left end toward the below-right end in thesubsequent processing, thus the new pixel position n is set.

At step 2904, the light source data deriving unit 205 acquires a depthvalue L (n) of the pixel position n in the depth image.

At step 2905, the light source data deriving unit 205 compares theacquired depth value L (n) and a threshold value determined in advance.As a result of comparison, in the case where the depth value L (n) islarger than the threshold value, the procedure proceeds to step 2906. Onthe other hand, in the case where the depth value L (n) is smaller thanthe threshold value, the procedure returns to step 2903 and the nextpixel position is set.

Here, the reason that processing is switched in accordance with thedepth value is explained. As light source data generally used in a CG,there is an infinitely distant light source map. The infinitely distantlight source map is a format to specify the luminance of a light sourceonly by the direction (latitude, longitude) on the assumption that thelight source is located at an infinite distance. In the case whereobeying this format, the use of a captured image having an image regionthat cannot be regarded as an infinitely distant image region for lightsource data may lead to a strong possibility that the result ofcombination is an unnatural one. Consequently, it is advisable to avoidthe use of the captured image including the image region that cannot beregarded as an infinitely distant image region for the light sourcedata. Because of this, based on the depth image, an image region notsuitable to the light source data is excluded from the captured image.FIG. 30A shows an example of a CG rendering result in the case where acaptured image including only subjects that can be regarded asinfinitely distant subject is used for light source data; and FIG. 30Bshows an example of a CG rendering result in the case where a capturedimage including subjects that cannot be regarded as infinitely distantsubjects is used for light source data, respectively. Then, on eachrendering result in FIGS. 30A and 30B, CG geometry that makes use oflight source data is shown, and the light source data surrounding thecube of the CG object is made visible and represented by a doom. From acomparing of both, in FIG. 30A, it is seen that the shadow of the cube,which is the CG object, is cast in the rendering result because lightenters between the tower and the buildings. The same shadow is cast inthe case where the real cube is placed in the real environment in whichlight source data is acquired, and therefore, it can be said that FIG.30A shows the CG rendering result representing reality. In contrast tothis, in FIG. 30B, the light that should enter between the buildings andthe tower is blocked by the person, and therefore, the shadow of thecube, which is the CG object, is not cast. That is, it cannot be saidthat FIG. 30B shows the rendering result representing reality. Further,the light source data represents the luminance of incidence light at theposition where image capturing is performed, therefore it is notpreferable for the data of the person not relating to the lightingenvironment at that position to be reflected in the light source data.It is preferable to enable for a user to vary the distance to beregarded as an infinite distance by the setting. For example, in thecase where the threshold value described above is set to 10 m, the lightsource data is generated by using the image region where the depth valueL(n) is larger than 10 m.

Explanation is returned to the flowchart in FIG. 29.

At step 2906, the light source data deriving unit 205 derives adirection (latitude, longitude) corresponding to the pixel position nfrom the image capturing condition data. FIG. 41 is a diagram showing arelationship between the pixel position n and a direction correspondingto the pixel position n. In this case, it is assumed that an opticalaxis direction is held in advance as image capturing condition data byacquiring the latitude and longitude using an electronic compass, gyro,horizon sensor, etc., built in a camera. Then, a direction correspondingto the pixel position n is found with the optical axis direction as areference. In order to simplify explanation, it is assumed that thecamera is placed horizontally. In this case, it is known from FIG. 41that a direction corresponding to the pixel position n the height ofwhich is the same as that of the image center is the optical axisdirection θn rotated horizontally. The angle θn is found by (pixeldeviation amount÷number of pixels)×field angle, where the field angle ofthe camera is taken to be θ. It is needless to say that information,such as the field angle and the number of pixels, is acquired from imagecapturing condition data.

At step 2907, the light source data deriving unit 205 sets a pixel valueof the captured image at the pixel position n for the direction(latitude, longitude) corresponding to the pixel position n derived atstep 2906.

At step 2908, the light source data deriving unit 205 determines whetherthe processing is performed for all the pixel positions of the capturedimage set to be subjected to processing. In the case where theprocessing is completed for all the pixel positions, the procedureproceeds to step 2909. On the other hand, in the case where there is apixel position not subjected to the processing yet, the procedurereturns to step 2903 and the next pixel position is set.

At step 2909, the light source data deriving unit 205 determines whetherthe processing is completed for all the captured images included in themulti-viewpoint image data. In the case where there is a captured imagenot subjected to the processing yet, the procedure returns to step 2902and the next captured image is set as the image to be subjected to theprocessing, and the processing at step 2902 to step 2908 is repeated. Onthe other hand, in the case where the processing is completed for allthe captured images, this processing is exited.

In the manner described above, light source data corresponding to animage capturing environment is generated from multi-viewpoint image dataand image capturing condition data.

In the present embodiment, for the sake of simplicity, the case wherethe number of images captured from different viewpoints is two isexplained as an example, however, as a matter of course, it is alsopossible to acquire more multi-viewpoint images and generate lightsource data by performing the processing described above using two ormore captured images.

Explanation is returned to the flowchart in FIG. 26.

At step 2605, the light source data output unit 206 outputs the lightsource data generated at step 2604. As the output method, it may also bepossible to record the light source data at the header of the image datato be saved or to generate and save a light source data file separatelyfrom the image file.

As described above, according to the invention of the presentembodiment, it is possible to derive light source data from data of acaptured image.

Seventh Embodiment

In the sixth embodiment in which light source data is derived byobtaining the depth image based on the parallax found from the pluralityof captured images, it is not possible to obtain a depth image in thecase where there is only one captured image. Here, an aspect isexplained as a seventh embodiment, in which a depth image is generatedfrom one captured image in a pseudo manner. Explanation of parts commonto those of the sixth embodiment is simplified or omitted and here,different points are explained mainly.

The flow of light source estimation processing in the present embodimentis similar to that of the flowchart in FIG. 26 according to the sixthembodiment; therefore explanation is given along the flow in FIG. 26.

At step 2601, the image data acquiring unit 201 acquires data of onecaptured image having been subjected to predetermined image processingfrom the signal processing unit 103. The acquired image data is sent tothe depth image generating unit 2501 and the light source data derivingunit 205.

At step 2602, the image capturing condition data acquiring unit 203acquires image capturing condition data from the operation unit 111 etc.The acquired image capturing condition data is sent to the depth imagegenerating unit 2501 and the light source data deriving unit 205.

At step 2603, the depth image generating unit 2501 generates a depthimage from the data of one captured image received from the image dataacquiring unit 201 and the image capturing condition data correspondingto the captured image.

As explained in the sixth embodiment, it is not appropriate to use thecaptured image including the image region that cannot be regarded as aninfinitely distant image region for the light source data as it is.Consequently, in the case where a depth image is generated from onecaptured image, it is also necessary to remove a subject located at ashort distance. In the present embodiment, in the case where a capturedimage includes a subject (here, a person) located within a specificdistance, a depth image is generated by regarding only the regions ofsubjects other than the subject as infinitely distant regions.Specifically, a depth image is generated as follows.

First, the depth image generating unit 2501 performs processing todetect the face region of the person for the captured image. To detectthe face region, a face detection technique widely known may be used.Face detection techniques include, for example, a method that makes useof pattern matching, in which partial images are cut out at a pluralityof different positions on the captured image, whether or not the partialimage is the image of the face region is determined, and thus, the faceregion on the captured image is detected.

Next, distance information is set by, for example, taking apredetermined region including the detected face region to be areference and by setting the distance to the outside of the referenceregion as a great distance (for example, 1 km) that can be regarded asan infinite distance and the distance to the inside thereof as a shortdistance (1 m), and so on, and thus, a depth image is obtained. In thiscase, the predetermined region may be set, for example, to a circularregion having an area double the area of the detected face region.

As described above, by detecting the face region in the captured imageand handling a fixed region including the detected face region as animage region at a short distance, a depth image is approximatelygenerated from one captured image. In the present embodiment, the caseis explained, where the person is detected as a subject located at ashort distance; however, for example, it may also be possible to detectan animal, such as a dog, a building, etc., by pattern matching etc. andto regard the outside of the detected animal or building as aninfinitely distant region.

At step 2604, the light source data deriving unit 205 derives lightsource data from the data of one captured image, the image capturingcondition data, and the depth image obtained in the manner describedabove. The light source data derivation processing in this case is thesame as that explained in the sixth embodiment except in that only onecaptured image is to be subjected to processing.

As a modification example of the sixth embodiment, it may also bepossible to add information indicative of a region used as light sourcedata to the captured image based on the detection result of the faceregion etc. without generating a depth image in a pseudo manner. In thiscase, at step 2905 of the flowchart in FIG. 29, whether the pixelposition n is in the set region is determined as a result, and in thecase where it is determined that the pixel position n is in the region,the procedure is caused to proceed to step 2906, and thus, light sourcedata is derived.

Eighth Embodiment

Next, an aspect is explained as an eighth embodiment, in which lightsource data is derived with high precision by excluding a cast shadowregion in a captured image. Explanation of parts common to those of theother embodiments is simplified or omitted and here, different pointsare explained mainly.

FIG. 31 is a function block diagram showing an internal configuration ofthe light source estimating unit 114 according to the presentembodiment. The light source estimating unit 114 differs largely fromthe light source estimating unit 114 according to the first embodiment(see FIG. 2) in that an exclusionary region determining unit 3101 isadded.

The exclusionary region determining unit 3101 performs processing toextract cast shadow regions in an image and determine a region of theextracted cast shadow regions that is in contact with the edge part ofthe image as an exclusionary region not used in light source estimation.

FIG. 32 is a flowchart showing a flow of light source estimationprocessing in the present embodiment.

At step 3201, the image data acquiring unit 201 acquires image datahaving been subjected to predetermined image processing from the signalprocessing unit 103. The acquired image data is sent to the light sourcedata deriving unit 205 and the exclusionary region determining unit3101.

At step 3202, the exclusionary region determining unit 3101 performscast shadow region extraction processing on the received image data.Methods for extracting a cast shadow region from a captured image arewidely known, and for example, it is possible to extract using the XYZvalues of the image. Specifically, after the RGB values are convertedinto chromaticities XYZ based on the used color space profile of thecaptured image, a pixel region whose luminance value Y<Y_(th) (forexample, Y_(th)=80) is extracted as a cast shadow region. There areother methods, such as the background difference method and a methodthat makes use of a change in lightness; however, explanation thereof isomitted here.

At step 3203, the exclusionary region determining unit 3101 determines apixel region not used in light source estimation (hereinafter, called an“exclusionary region”) of the cast shadow regions extracted at step3202. Whether a region is regarded as an exclusionary region isdetermined by checking whether the extracted cast shadow region is incontact with the edge part of the captured image and by regarding theshadow as cast by an object not included in the image in the case wherethe cast shadow region is in contact with the edge part of the image.FIG. 33 is a diagram showing an example of an exclusionary region. InFIG. 33, two cast shadow regions 3301 and 3302 are shown and the castshadow region 3302 is in contact with the top-left edge part of theimage. In this case, the cast shadow region 3302 is determined as anexclusionary region as a result.

Here, the reason that light source data can be derived with highprecision by excluding a cast shadow region at the edge part of an imageis explained. FIG. 34 is a diagram showing a way the image of a scene,in which a cuboid and a cube arranged on a floor are irradiated withlight having a luminance of 1 (lux) from the direction of a point at 45°latitude and 0° longitude, is captured by a camera. Now, a case isconsidered where light source data is derived from the image captured insuch a state. Here, it is assumed that reflection is Lambert reflectionin which the luminance of light reflected from the floor, the cuboid,and the cube is uniform with respect to the emission direction and thereflectance is 1. In the present embodiment, in order to simplifyexplanation, a case of a white-and-black image with one channel isexplained; however, in a case where a color image is used, it issufficient to perform the same processing for each of the three channelsof R, G, and B. FIG. 35A shows a captured image including a cube and itscast shadow at the center of the image and a shadow cast caused by acuboid at the top-left part of the image. First, a case is consideredwhere light source data is derived using only the cast shadow regionproduced by the cuboid of the pixel region of the captured image. Inthis case, there is Lambert reflection from the floor and the pixelvalue in a cast shadow region A is 0, and therefore, light source datathat is derived will be light source data in which light intensity is 0(lux) in all the latitude and longitude directions (see FIG. 35B). Next,a case is considered where light source data is derived using the entireregion except for the cast shadow region of the pixel region of thecaptured image. At this time, in the light source estimation processingexplained in the first embodiment (see the flowchart in FIG. 9), lightsource data is derived, with which the error Δ_(i) between the CG imageand the captured image, in which the cube is irradiated with lighthaving a luminance of 1 (lux) from the direction of a point at 45°latitude and 0° longitude, will be smaller than a predeterminedthreshold value. That is, the light source data that is derived will belight source data in which the light intensity in the direction of apoint at 45° latitude and 0° longitude is 1 (lux) as shown in FIG. 35C.As described above, it is known that the light source data obtained fromthe cast shadow region and the light source data obtained from theentire region except for the case cast shadow region differ largely fromeach other. FIGS. 36A and 36B show light source data that is derived inthe case where there is no region-to-be-excluded and in the case wherethere is a region-to-be-excluded. FIG. 36A shows the light source datathat is derived in the case where there is no exclusionary region (inthe case where light source estimation is performed using the entirepixel region in the image), light source data in which the lightintensity in the direction of a point at 45° latitude and 0° longitudeis 0.8 (lux). On the other hand, FIG. 36B shows the light source datathat is derived in the case where there is an exclusionary region (inthe case where light source estimation is performed using the pixelregion excluding the cast shadow region in the captured image), lightsource data in which the light intensity in the direction of a point at45° latitude and 0° latitude is 1.0 (lux). As is obvious from comparisonof both, the light source data of the image capturing scene is derivedwith higher precision in the case where there is an exclusionary region.

In the present embodiment, the cast shadow region including the edgepart of the captured image is regarded as a shadow cast by an object notincluded in the captured image and determined as an exclusionary region;however, it may also be possible to apply the already existing objectrecognition technique to the captured image and to determine the castshadow region not in contact with the subject recognized as a solid asan exclusionary region. Further, it may also be possible to enable auser to specify a region that the user does not intend to use in lightsource estimation on a UI.

The processing at subsequent step 3204 to step 3206 is the same as thatat step 502 to step 504 of the flowchart in FIG. 5 according to thefirst embodiment, and therefore, explanation thereof is omitted here.

At step 3207, the light source data deriving unit 205 derives lightsource data most suitable to the scene based on the acquired image data,the subject shape data, the image capturing condition data, and thedetermined initial light source data. The light source data derivationprocessing in the present embodiment is also basically the same as thatof the flowchart in FIG. 9 according to the first embodiment; however,the processing at step 903 is different. That is, in the presentembodiment, in the case where the error Δ_(i) between the CG image andthe captured image is found, the error Δ_(i) from the CG image is foundusing the pixel region except for the exclusionary region (the pixelregion, which is the entire pixel region of the captured image fromwhich the exclusionary region is excluded) as a result.

At step 3208, the light source data output unit 206 outputs the derivedlight source data.

As described above, according to the invention of the presentembodiment, it is possible to derive light source data with higherprecision by excluding a cast shadow region in a captured image and bynot using the cast shadow region in light source estimation.

Ninth Embodiment

In the eighth embodiment, the cast shadow region in contact with theedge part of the image of the cast shadow regions included in thecaptured image is determined as the exclusionary region not used inlight source estimation. However, in a shooting scene in which lightenters obliquely, such as a sunset scene, or in a shooting scene inwhich a main subject is included at the edge part of the image, etc.,the shadow cast by the main subject included in the captured image alsocomes into contact with the edge part of the image, therefore it isdifficult to appropriately determine an exclusionary region. Because ofthis, an aspect is explained as a ninth embodiment, in which anexclusionary region is determined using initial light source data sothat it is possible to appropriately determine an exclusionary regionalso in the image captured in such a shooting scene described above.Explanation of parts common to those of the other embodiments issimplified or omitted and here, different points are explained mainly.

FIG. 37 is a function block diagram showing an internal configuration ofthe light source estimating unit 114 according to the presentembodiment. The light source estimating unit 114 differs largely fromthe light source estimating unit 114 according to the eighth embodiment(see FIG. 31) in arrangement (relationship to other units) of anexclusionary region determining unit 3601. That is, the exclusionaryregion determining unit 3601 in the present embodiment acquires not onlyimage data from the image data capturing unit 201 but also subject shapedata, image capturing condition data, and initial light source data anduses to determine an exclusionary region.

FIG. 38 is a flowchart showing a flow of light source estimationprocessing in the present embodiment.

At step 3801, the image data acquiring unit 201 acquires image datahaving been subjected to predetermined image processing from the signalprocessing unit 103. The acquired image data is sent to the light sourcedata deriving unit 205 and the exclusionary region determining unit3601.

At step 3802, the subject shape data acquiring unit 202 acquires subjectshape data. The acquired subject shape data is sent to the light sourcedata deriving unit 205 and the exclusionary region determining unit3601.

At step 3803, the image capturing condition data acquiring unit 203acquires image capturing condition data from the operation unit 111 etc.The acquired image capturing condition data is sent to the initial lightsource data determining unit 204, the light source data deriving unit205, and the exclusionary region determining unit 3601.

At step 3804, the initial light source data determining unit 204determines initial light source data corresponding to the acquired imagecapturing condition data by referring to a light source data table savedin the ROM 108 etc.

At step 3805, the exclusionary region determining unit 3601 performsrendering based on the acquired image data, the initial light sourcedata, the subject shape data, and the image capturing condition data togenerate a CG image. FIG. 39 is a diagram showing an example of a CGimage generated by rendering. The shadow cast caused by the cube locatedat the center of the screen is in contact with the below-right edge partof the image.

At step 3806, the exclusionary region determining unit 3601 performscast shadow region extraction processing on the captured image acquiredat step 3801 and the CG image generated at step 3805, respectively.Extraction of the cast shadow region is the same as that explained inthe eighth embodiment (see step 3202 of the flowchart in FIG. 32).

At step 3807, the exclusionary region determining unit 3601 calculates aratio P of the area of the cast shadow region that overlaps that of theCG image to the area of the cast shadow region for each cast shadowregion in the captured image. Then, the exclusionary region determiningunit 3601 determines whether the calculated ratio P is smaller than apredetermined threshold value P_(th) (for example, P_(th)=0.5) anddetermines the cast shadow region whose P satisfies P<P_(th) as anexclusionary region not used in light source estimation. FIG. 40 is adiagram for explaining the process to determine an exclusionary regionin the present embodiment. In FIG. 40, the ratio P of the area of a castshadow region 4001 in the captured image that overlaps a cast shadowregion 4003 in the CG image to the area of the cast shadow region 4001is 0.85. On the other hand, there is no cast shadow region in the CGimage that corresponds to a cast shadow region 4002 in the capturedimage, therefore the ratio=0. Consequently, in this case, although thecast shadow region 4001 in the captured image is in contact with thebelow-right edge part of the cast shadow region 4001 in the capturedimage, the calculated ratio P is larger than the threshold value P_(th)(0.5), and therefore, the exclusionary region determining unit 3601 doesnot determine that the cast shadow region 4001 is an exclusionary regionand only the cast shadow region 4002 is determined as an exclusionaryregion.

At step 3808, the light source data deriving unit 205 derives lightsource data optimum for the scene based on the acquired image data, thesubject shape data, the image capturing condition data, and thedetermined initial light source data. The light source data derivationprocessing in the present embodiment is the same as that at step 3207 ofthe flowchart in FIG. 32 in the eighth embodiment. That is, in the casewhere the error Δ_(i) between the CG image and the captured image isfound, the pixel region other than the exclusionary region (the entirepixel region of the captured image from which the exclusionary region isexcluded) is used to find the error Δ_(i) from the CG image as a result.

At step 3809, the light source data output unit 206 outputs the derivedlight source data.

As described above, according to the invention of the presentembodiment, it is made possible to derive light source data with highprecision even from an image captured in a shooting scene of a sunsetwhere light enters obliquely, or a captured image in which a subject islocated at the edge part.

Tenth Embodiment

In the first embodiment, the aspect is explained, in which imagecapturing condition data is used in the case where the initial value forcalculation of light source estimation is determined. Next, an aspect isexplained as a tenth embodiment, in which in a case where a plurality ofsolution candidates is found in the calculation of light sourceestimation, one piece of light source data is determined by using imagecapturing condition data. Explanation of parts common to those of theother embodiments is simplified or omitted and here, different pointsare explained mainly.

FIG. 42 is a diagram showing a system configuration example of aninformation processing system for performing light source estimationaccording to the present embodiment. A light source estimation device4200 is a PC etc. and substantially corresponds to the light sourceestimating unit 114 in the first embodiment. The light source estimationdevice 4200 according to the present embodiment includes an optimumlight source data determining unit 4201. The optimum light source datadetermining unit 4201 determines one piece of light source data based onimage capturing condition data from a plurality of solution candidatesderived by the light source data deriving unit 205.

FIG. 43 is a flowchart showing a flow of light source estimationprocessing in the present embodiment.

At step 4301, the image data acquiring unit 201 of the light sourceestimation device 4200 acquires image data having been subjected topredetermined image processing from a camera etc., not shownschematically. The acquired image data is sent to the light source dataderiving unit 205.

At step 4302, the subject shape data acquiring unit 202 of the lightsource estimation device 4200 acquires subject shape data from an HDD(not shown schematically) etc. in which subject shape data is stored.

At step 4303, the image capturing condition data acquiring unit 203 ofthe light source estimation device 4200 acquires image capturingcondition data from a camera etc., not shown schematically. The acquiredimage capturing condition data is sent to the light source data derivingunit 205.

At step 4304, the light source data deriving unit 205 derives lightsource data based on the subject shape data. An estimation algorithmused at this time derives a plurality of solution candidates. Forexample, in the case where solutions are found by updating light sourcedata as described in the formula (3), for example, it can be thought touse the genetic algorithm. In this case, a plurality of solutions issearched for and each of the solutions is a solution candidate of lightsource data.

At step 4305, from the plurality of solution candidates obtained at step4304, one solution candidate is determined as optimum light source data.This determination makes use of, for example, light source datadescribed in FIGS. 7A to 7D. The optimum light source data determiningunit 4201 determines a solution candidate closest to the light sourcedata described in FIG. 7A as optimum light source data from theplurality of solution candidate for the data of an image captured withthe WB (cloudy). At the time of the determination of the closestsolution candidate, a squared error is determined as an evaluationvalue.

At step 4306, the optimum light source data determined at step 4305 isoutput.

As described above, in the present embodiment, in the case where aplurality of solution candidates is found in the calculation of lightsource estimation, it is made possible to determine one piece of lightsource data using image capturing condition data.

In the present embodiment, one solution candidate is selected as optimumlight source data from a plurality of solution candidates of lightsource data; however, for example, it may also be possible to assign anappropriate weight to each of the plurality of solution candidates andadd them. That is, it is made possible to derive more preferred optimumlight source data by determining a weight coefficient based on imagecapturing condition data and adding the plurality of solutioncandidates.

Other Embodiments

Aspects of the present invention can also be realized by a computer of asystem or apparatus (or devices such as a CPU or MPU) that reads out andexecutes a program recorded on a memory device to perform the functionsof the above-described embodiment (s), and by a method, the steps ofwhich are performed by a computer of a system or apparatus by, forexample, reading out and executing a program recorded on a memory deviceto perform the functions of the above-described embodiment (s). For thispurpose, the program is provided to the computer for example via anetwork or from a recording medium of various types serving as thememory device (e.g., computer-readable medium).

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application Nos.2012-108822, filed May 10, 2012, and 2013-022565, filed Feb. 7, 2013,which are hereby incorporated by reference herein in their entirety.

What is claimed is:
 1. An information processing device comprising: animage capturing condition data acquiring unit configured to acquireimage capturing condition data at the time of capturing an image of asubject; and a light source data deriving unit configured to derivelight source data which represents a state of a light source of an imagerepresented by the data of the captured image based on the imagecapturing condition data.
 2. The information processing device accordingto claim 1, wherein the light source data deriving unit has adetermining unit configured to determine an initial value at the time ofestimating the light source based on the image capturing condition data.3. The information processing device according to claim 2, furthercomprising a shape data acquiring unit configured to acquire shape datawhich represents shape of the subject, wherein the light source dataderiving unit performs rendering using shape data acquired by the shapedata acquiring unit.
 4. The information processing device according toclaim 2 further comprising a subject shape estimating unit configured toestimate shape of the subject from multi-viewpoint data of images of thesubject captured from a plurality of viewpoints, wherein the lightsource data deriving unit performs rendering using subject shape dataobtained by the estimation.
 5. The information processing deviceaccording to claim 3, wherein the light source data deriving unitderives light source data suitable to the captured image based on anerror between an image obtained by the rendering and the captured image.6. The information processing device according to claim 5, wherein thelight source data deriving unit derives light source data with which anerror between an image obtained by the rendering and the captured imagebecomes smaller than a threshold value as light source data suitable tothe captured image.
 7. The information processing device according toclaim 6, wherein the threshold value is changed in accordance with anevaluation value which represents accuracy of subject shape dataestimated by the subject shape data estimating unit.
 8. The informationprocessing device according to claim 2, wherein the light source dataderiving unit performs rendering using simplified shape data in place ofthe subject shape data and derives light source data suitable to thecaptured image based on an error between an image obtained by therendering and the captured image.
 9. The information processing deviceaccording to claim 1, wherein light source data corresponding to theimage capturing condition data is determined by referring to a lightsource data table including a plurality of pieces of light source dataassociated with image capturing condition data.
 10. The informationprocessing device according to claim 1, wherein each piece of lightsource data included in the light source data table is specified by avalue which represents luminance for each incidence direction of lightincident on a subject.
 11. The information processing device accordingto claim 10, wherein the light source data table includes an evaluationfor precision of each piece of light source data.
 12. The informationprocessing device according to claim 1, further comprising a featurequantity deriving unit configured to derive a feature quantity whichrepresents a feature of an image represented by the captured image datafrom the captured image data, wherein the light source data derivingunit derives light source data corresponding to the image capturingcondition data and the derived feature quantity as light source datawhich represents a state of a light source.
 13. The informationprocessing device according to claim 1, further comprising a depth imagegenerating unit configured to generate a depth image which representsinformation of a distance to a subject through a plurality of pieces ofmulti-viewpoint image data which consists of images captured from aplurality of different viewpoints acquired by the image data acquiringunit, and the image capturing condition data, wherein the light sourcedata deriving unit derives light source data which represents a state ofa light source by excluding an image region not suitable as a lightsource from each image within the images of the plurality of pieces ofmulti-viewpoint image data based on the depth image.
 14. The informationprocessing device according to claim 1, further comprising anexclusionary region determining unit configured to determine a specificregion within the captured image as an exclusionary region not used forlight source estimation, wherein the light source data deriving unitfinds an error from an image obtained by the rendering using a pixelregion, which is the entire pixel region of the captured image fromwhich the exclusionary region is excluded, and derives light source datasuitable to the captured image based on the error that is found.
 15. Theinformation processing device according to claim 1, further comprisingan exclusionary region determining unit configured to determine aspecific region within the captured image as an exclusionary region notused in light source estimation based on an image obtained by performingrendering using shape data of a subject within the captured image,wherein the light source data deriving unit finds an error between thecaptured image and an image obtained by the rendering, using a pixelregion which is the entire pixel region of the captured image from whichthe exclusionary region is excluded, and derives light source datasuitable to the captured image based on the error that is found.
 16. Acamera including the information processing device according to claim 1.17. An information processing method comprising the steps of: acquiringimage capturing condition data at a time of capturing a subject;inputting shape data which represents shape of the subject; and derivinglight source data which represents a state of a light source of an imagerepresented by the captured image based on the image capturing conditiondata and the shape data.
 18. A program stored in a non-transitorycomputer readable storage medium for causing a computer to perform theinformation processing method according to claim 17.